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INTRODUCTION 

i 

A central focus of the CSE project relating to evaluation activities 
in school districts is the question of the extent to which school district 
evaluation offices are contributing to the renewal of public 'education. 
Part of this question c*n be answered empirically by examining data which 
describes how local evaluation ageifcies^^ are organized and staffed 
and what functions they serve (Lyon^et al, 1978). Still another ps|rt of 
this larger question can be addressed through organizational analyses using 
survey data to test propositions concerning the impact of structure or con- 
flict (cf. on evaluation activities, Grusky, 1980; Zucker. 1980L As 
David (1978) has recently documented,, reappears clear that Title I eval- 
uation results do not serv$, primarily, as a means of judging -the effec- 
tiveness of a program or as a guide to further program improvement; that 
evaluation information is rarely seen as Important in the context of de- 
cisions about program changes. David goes on to point out that "literature 
on evaluation has only recently included attempts to understand the role 
and use of evaluation results... in the realm of decision making" (1978, 
p. 8). Her study, in fact, provides evidence of the almost total lack 
of utilization of evaluation data by decision makers in school districts. 

Since the CSE survey data does* riot directly address the empirical 
testing of the propositions generated here, this paper may best be seen 
as art attempt to render the reader sensitive to some .factors which may 
preclude evaluation information as a force for change. From this per- 
spective, locaV evaluators may be able to anticipate what aspects of 
their evaluation will be seen as convincing or useful by various decision 
making audiences. 




BACKGROUND 

In the past several years* evaluation researchers and practitioners 
have become concerned about the apparent failure to utilize evaluation 
Information. For instance, Davis and Salasin (1975) acknowledged that 
even methodologically sound research 1s often not used. Because many 
people accept the* proposition that the purpose of evaluation research 1s 
to produce Information primarily for use by decision makers, their concern 

about a 1 lack of utilization of evaluation findings by decision makers ad- 

\ 

dresses a fundamental Issue as it calls Into question the very purpose of 
the evaluation function. 

No true consensus can be said to exist about whether this perceived 
lack of utilization is accurate. There is evidence both supporting and 
attacking the claim of non-utilization. For example, a number of authors 
have noted that the Head Start program continues „ in spite of the results 
of almost ten years of research demonstrating that it has had almost no 
effect on cognitive abilities (e.g., Goldstein, Marcus, and Rausch,. 1978). 

•n 

Similar instances of v lat appears to be non-utilization can also be found: 
Ward and Kassebaum (1972), for eraaiple, report that program administrators 
ir? a state correctional system counseling program systematically Ignored 
evcilua^: n data showing that the program was ineffectual with respect to 
the specified outcomes, decided to extend the^ program and, in certain cases 
to make it mandatory. 

A number of authors have catalogued possible explanations for such 
situac - r . 

1. A lack of rigor 1n the chslgn and conduct of the evaluation 
study (e.g., Goldstein et al, 1978). 



2. Too much rigor so that the research falls to assess the 
true purpose of the program .(e.g. , Cox, 1977). 

3. Studies which provide outcomes which aren't helpful (such 
as negative findings), results which aren't timely enough, 
or reports which are too lengthy and filled with jargon 
(Alkln, Cox, 1977; Goldstein et al, 1978). 

4. Communication problems between evaluators and program 
administrators (Cox, 1977; Goldstein et al, 1978). 

Arrayed against these arguments and examples are counter- examples • 
and claims that utilization does, 1n falct, occur. Patton (1978), 1n a 
survey of federal evaluators, reported that 78 percent of those surveyed 
felt that evaluation studies had an Impact on their programs. He, and 
others, argue that a broader definition of "utilization" must be 1n order 
properly to assess the Impact of evaluation research. When the definition 
of 'utilization fs extended to Include Impacts such. as providing additional 1 - 
credibility for a program, verifying managers' suspicions, and generally 
providing additional information, then, 1t 1s argued, the utilization of 
evaluation Information 1s more widespread than 1s commonly believed (Levlne 
& Levlne, 1377). The problem, 1n this view, 1s primarily that the def1n1t1o 
of "utilization" 1s too narrow and "overly rational" (Patton, 197-8, p. 40). 

Essentially, whether utilization 1s or 1s not problematic depends on„ 
the definition of the term, on what 1s meant by "rational," and on the 
evidence the discussant uses 1n arriving at a conclusion; on what Infor- 
mation the* discussant seeks out or is exposed to, how closely he or she 
chooses to attend to the various arguments, and whether he or she has a 
vested Interest 1n the outcome of the dispute. From this- perspective, ul- 
timately the problem of utilization or non-utilization of evaluation 



Information 1s one of organizational processes and pressures which en- 
courage or discourage the production, transmission, and use of Informa- 
tion by.' relevant decision makers. 

The purpose of this paper 1s to analyze the utilization of evaluation 
research from an organizational behavioral perspective. To accomplish 
this, we will present a simplified Information-processing model of decision 
making. With this orientation, the utilization or non-ut1Hzat1on of eval- 
tlon Information may be seen as a function of two primary factor?: (1) 
the organizational context 1n which decision makers function, Including 
an awareness of operating goals and Incentives, .and (2) the Information 
processing constraints on the decision makers, Including availability of 
Information, costs of search, and the manner 1n which. Information 1s per- 
ceived and - processed. Drawing on the research literature related to infor- 
mation use and organizational decision making, a series of propositions^ will 
be generated that .suggest conditions under which evaluation Information 1s 
likely O to be used, or not used, by decision makers. \ 

AN INFORMATION PROCESSING MODEL FOR DECISION MAKING 

Decision making is, simply put, the act of choosing among alternatives. 

Ideally, the decision maker 1s presumed to have a set of values or evaluative 

criteria, the perception of a problem which requires action', a number of 

potential alternative solutions, an.d a "calculus for comparing alternatives 

and estimating the likelihood of attaining certain outcomes given certain 

alternatives. With perfect rationality, the assumptions leading tochoice 

; ■ ' •• ■ . v ' • j ■ 

generally include complete information about the alternatives,, knowledge 



of the probabilities associated with different alternative-outcome links, 
a consistent ordering of preference among outcomes, and a selection mechanism 
which maximizes the value attained by a choice. However, viewed against 
the conflict of bounded rationality, the decision maker 1s seen as Intendedly 
National but acting with limited computational abilities. In both cases, 
however, the process of choice 1s one of Information assimilation and use, 
suggesting that an Information processing model may be a fruitful way to 
examine the decision making process. 

Consider the simplified model of decision making presented in Figure 
1- In this schema, the decision maker 1s confronted with a problem or 
situation requiring a choice, generates potential alternative solutions, 
assesses the probabilities that a given alternative will lead to certain 
outcomes, and develops a preference ordering among outcomes. In this view, 
information and the ability to process it are paramount. Information pro- 
cessing 1s required in all phases, i.e., to define the problem, develop 
alternatives, estimate probabilities, and order outcomes. 

Insert Figure 1 here 



The emphasis in this model is not that it is a complete or literal 
representation of the decision making process. A variety of similar models 
havebeeh proposed (.i.e., Cohen, March, & Olsen, 1974: Janis & Mann, 1977; 

• 3 

MacCrimmon, 1974), and evidence is available which suggests that such models 
may be oversimplifications (Witte, 1972). Mintzberg, Ra1s1nghami, and 
Theoret (1976), for example, traced 25 complex organizational decisions 

_ ■ * . ■ 
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and characterize the process as a plurality of sub-decisions without a 
simple sequential relationship such as indicated In Figure I. Neverthe- 
less, the model 1n Figure 1 1s useful 1n that 1t emphasizes the central ity 
of Information and Information processing 1n the decision process. It 
also suggests a framework for organizing and investigating the limitations 
and constraints on organizational decision making. 

In a typical formulation, decision makers encounter or are presented 
with problems which require choices, e.g., continue *funding for a project 

m 

or reallocate funds to other purposes, or allow the hiring of additional 
staff. The decision maker then considers a variety of alternatives for 
solving the problem. Each alternative 1s examined and some subjective 
weight or probability estimated that selection of given alternatives will 
lead to particular outcomes. Typically, the decision maker 1s not indif- 
ferent among the set of outcomes but has preferences reflected by weights 
for at least some of the outcomes. Given that Information processing limits 
exist, 1t nay be that the' decision maker will not have complete knowledge 
of the alternatives, probabilities or outcomes. In general, however, the 
decision making process 1s presumed to operate from left to right. Under 
the traditional assumptions of rationality ascribed to evaluation researchers 
it 1s assumed that the decision maker will search for unbiased information 
about the .various, components" in the model, and that the weights attached 
to various outcomes are determined by organizational goals; that 1s, the 
decision maker will attempt to make the choice such that the ..maximum net 
benefit accrues to the organization or the agency. 

But is this left-to-right progression an accurate description of how 



decisions are made? Several authors have proposed that, In organizations, 
problems are seldom clearly defined and alternatives become known only 
after certain outcomes were preferred (Cohen et al, 1974; Welek, 1977), 
It may be, 1n fact, that In organizations, the decision making process 
begins with the preference ordering for outcomes as a set of rather fixed 
constraints. Simon (1964) suggested fthls when he proposed that organiza- 
tional goals be viewed as constraint sets. Certainly, 1n organizations 
Individuals and subunlts typically have vested Interests. The loss of 
resources, status, and power are to be avoided, Even the uncertainty 
caused by change may be resisted, increases In resources, for example, 
more funding or more staff, are typically preferred. Decision makers are 
Indifferent about outcomes only when they are unaffected. In fact, de- 



cision makers may become participants 1n a decision making process to 1n- 



The fact that organizational decision makers h&ve strong preferences 
for certain outcomes 1s well documented (e.g., Pfeffer & Salandk, 1977,** 
1978). These preferences, discussed at greater length later 1n this paper 
reflect not only organizational and subunlt goals, but also individual 
desires and concerns such as promotion andgSalary. Because these prefer- 
ences exist, 1t follows that managers or administrators are unlikely to 
remain passive when decisions are to be made which affect them. Conflicts 
may develop when decisions can result 1n gains and losses to the partici- 
pants. Under these circumstances, alternatives which lead to undeslred 
outcomes may* become unacceptable to certain participants. In addition, 
as Thompson (1967) observed, uncertainty threatens rationality. Managers, 
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sure that they remain unaffected. 
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iipteliUy those 1n power, attempt to eliminate potentially disruptive 
uncertainty, information, In this context, becomes a political resource. 
Pata which supports desired outcomes Is sought out, while information 
which supports opposite views is to be rebutted by questioning its ac- 
curacy, for example, by obtaining other Information which supports a 
countervailing view, or by impugning the credibility of the source of 
the threatening information, Information, then, is a commodity used for 
a variaty of purposes. Under some circumstances it may be used as. a basis 
for decision making-, 1n others, as corroboration for decisions already 
made, and 1ri still others, for symbolic reasons. Information, 1n this 
milieu, 1s not a fixed substance, but one which may be selectively pm 
celved and processed. 

Thus, the argument being made 1s twofold. First, that in organiza- 
tions, participants typically have preferences for outcomes which reflect 
organizational as well as Individual goals. These outcomes act to define 
a set of constraints which result 1n the decision process moving from right 
to left as portrayed In Figure 1. Second, that because of the potential 
for disagreements among participants, the entire decision process may be 
one of bargaining and negotiation as various actors attempt to pursue their 
interests. In this political process, information becomes a potentially 
important, or threatening, commodity. It 1s generally not perceived of 
as "objective." Rather, depending on the nature and importance of the 
goals sought, decision makers may systematically search for supporting 
information while ignoring other types. 

It is this organizational context 1n which evaluation research and 



evaluation units whether at local, state or federal level, should be ex- 
amined. Both the evaluation unit and the information 1t produces may be 
assets or liabilities for decision makers, ' 

Evaluation information may be categorized 1n several ways. For ex- 
ample, one can distinguish between formative and summatlve evaluation, 
with the former referring to information focused on improving the work- 
ings of a given program and 1% latter morfe concerned with overall impact 
or effectiveness of a given project. Formative evaluations are typically 
used to provide information to administrators to improve the conduct of 
the program, and, as such, may be less threatening than a summatlve eval- 
uation which may dictate the continuation or discontinuation of the entire 
project. We can make still another distinction between Information col- 
lected for routine compliance with funding agency guidelines and* non-routine 
information used to assess goal attainment. The former information is 
often* critical in that continued funding is predicated on demonstrated com- 
pliance and substantial efforts and resources are often rf qui red to collect 
such Information. This Information, however critical, 1s not typically 
used to assess program performance. For the purposes of this paper, "eval- 
uation Information" or "evaluation research" will refer primarily to the 
more non-routine summatlve aspects of evaluation research. While some 
of the material developed may be relevant to an understanding of routine 
or formative ^valuation, "the primary focus here is on information about non- 

f • 

routine or summatlve events. * 

We propose to discuss, the concept of "rationality" as it applies to 
organizational decision making and Information use, the Impact of 
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organizational goals and Incentives on the'model "present in Figure 1, and 
finally, drawing upon a diverse body of research on cognitive information ■ 
processing and decision making, a number of potential biases in Information 
processing which may affect the use of evaluation information are explored. 

RATIONALITY AND THE DECISION MAKING PROCESS 

As indicated previously, the notion of rationality implied by Figure 
1 1s not the # comprehensive rationality of economic theory in which unbiased 
decision makers use perfect information to maximize utility according to 
some completely specified and ordered preference set. Instead t as sug- 
gested in Figure 1, decision makers begin with preferences and select 
actions based on imperfect expectations about the effect of these actions 
upon future preferences and thai? these actions may include limited and 
focused information search and selective perception and processing of in- 
formation, Thus, "rational 1ty," as used here, does not even refer solely 

* ***<>, 
-to the notions of bounded rationality 1"n 'which choices are made by. decision 

makers who use Imperfect 'information 1n a satisfying manner to maximize 

« 

goal attainment subject to actual rather than perfect knowledge (March & 
Simon, 1958). Instead, the appropriate calculus for Figure 1 appears to 
be one which March (1978,,*p. 592) refers to as "contextual rationality" 
1n which choice behavior 1s embedded 1n a complex of other claims on the 
attention of decision makers and other structures of. social and cognitive 
relations. Organizational decision makers, in this view, are pursuing 
multiple objectives subject to a variety of pressures and constraints, 
and often With considerable ambiguity surrounding the choice process. • 



Under these circumstances, preferences for outcomes may be the, least 
'ambiguous component' of the decision process, more certain than the de- . 
f1n1t1on of the problem, the range of feasible alternatives, or the prob- 
abilities associated with various alternatives. In this situation, it is 
argued that decision makers are likely to take action which both reduces 
their uncertainty and helps them achieve desired outcomes (e.g.l, search 
for supportive information or selectively interpret signals as favorably 
to a preferred outcome). Lindblom (1959) offers some support for this 
when he observed that the selection of goals and the empirical analysis 
of the needed actions to obtain the goals are not distinct from one an- 
other but are closely intertwined. Since ends typically come before means 
a:nd may be known with greater certainty, it is likely that the search for 
appropriate means will be highly focused. 

Since Individual decision makers are known to be limited in their 
ability to solve problems, organizational routines are often established 
to increase -the likelihood that individuals will behave in a traditionally 
rational way. This focus 1s on the effectiveness of procedures usted to 
make choices,, what Simon refers to as "procedural rationality" (1978, p. 8) 
These -routines are established, 1n light of the limitations suggested by 
bdunded and contextual rationality, to emphasize rational search procedures 
and, Insofar as possible, to ensure that decision makers have complete 
Informatibtk • This procedural rationality may easily Include provisions 
for information and control systems, carefully prescribed .review processes, 
and even mandated program evaluations. The explicit attempt is to ensure 
that the organizational content promotes art approximation to comprehensive 

• 10 



rationality by decision makers. These manifestations of procedural ra- 
tlonal-ity may be of considerable symbolic importance (e.g., Meyer &£owan, 
1977); but o' limited efficacy 1f decision makers are only contextually 
rational. V 

Such a reformulation of traditional notions of rationality is hardly 

new. For a number of years economists, political scientists and others 

have acknowledged that "perfect rationality" is not an apt description 

of real-world decision making (e.g., Etzioni, 1967; Floden & Welner, 1978; 

« - * * 

Gershuny, 1978; Lindblom, 1959). What is, perhaps, relevant here is that 

some of the disillusionment with social science evaluation" can be partly 
attributed to at least an Implicit acceptance of the assumptions under- 
lying traditional rationality. It 1s worth re-emphas1z1ng that in an or- 
ganizational context, unlike what Is Implied 1n the traditional rational 
model, goals are often 1ll-spec1f1ed or lack consensus, Information may 
be Incomplete, ambiguous and Imperfect, and decision makers may be pursuing 
multiple-competing objectives and lacking the time and -computational abili- 
ties necessary to adequately utilize "the available Information. It may 
even be, as suggested here, that the decision Itself may be driven by the 
solution rather than the problem'. Acknowledging these .constraints, *let' 
us now turn our attention tot (1) some of the relevant organizational 
properties alluded to by the notion of contextual rationality, and (2) 
the cognitive processing, limitations suggested'by the notion of bounded 
rationality. 

< . ,. . - 
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ORGANIZATIONAL IMPACTS ON DECISION MAKING 

Organizational Power and Evaluation 

To understand the Impact, of contextual influences on evaluation re- 
search, let us first consider the evaluation unit in an organizational 
context; that is, consider the evaluation unit as one group among many 
subuni-ts. As Pfeffer and Salancik (1978) have noted, suburiits -within or- 
ganizations may be viewed as actors competing 'for resources in a political 
arena. Power, in this milieu, becomes a critical determinant of the 

unit's ability tb acquire scarce and critical resources (Pfeffer & Salan- 
cik, 1974). . . ' 

Power, the crucial ingredient in this contest, may be envisioned as 
resulting from the subunit's ability to control critical contingencies; 
that is,. power stems from the ability to reduce the primary uncertainty 
which faces the organization. As formulated by Hickson f et al (1971), 
power or the ability to control contingencies and. reduce uncertainty, may 
vary according to: (1) how critical the uncertainty is which the subunit 
can reduce, that is, how central the contingencies are for the overall 
functioning of the orga/rTzatlon, (2) how effective the subunit Is In re- 
ducing the uncertainty, and. (3) how "substltutable" the function served by 
the subunit' 1s, that 1s, 'how easily the uncertainty reduction can be obtained 
from other sources within the organization. .Subunits that are successful 
according to these criteria are likely to be powerful. Those units that 
do not act to reduce Important uncertainty for the organization, or whose 
ability to reduce uncertainty 1s not uniquely held, are unlikely to exert N 
substantia,] Influence with the organizations. 

V 
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Subunit .power has important ramifications for an understanding of 
the utilization of evaluation information for several reasons. First, 
Pfeffer and Salancik (1974) have shown that power is directly related to 
a subunit 1 s ability to obtain scarce and critical resources, for example, ' 
budget allocations and staff. Without these, it may be difficult for an 
evaluation unit to perform effective evaluations. Second, Pfeffer and 
Salancik (1977) have also demonstrated that power is used to define" the 
criteria used jn decision i making.^ Mo re powerful subunits'may successfully 
emphasize those criteria on which they compare favorably. This ability 
to define or specify evaluative criteria may ^have di rect impacts on pro- 
gram evaluation, allowing powerful programs to either specify the basis 
on which they are to be evaluated.and. thereby fhsuring their continued 
success, or by redefining a posteriori the evaluative criteria to invalidate 
unfavorable findings. Without power, members of the evaluation unit are 
likely* to find that they lack, discretion in selecting the criteria and 
accepting others' definitions (e.g.-, Meltsner, 1976), or that they adhere 
to the more universalis tic standards of research only to have their product 
labelled as irrelevant or ignored by decision makers.' 

A final ramification of power which has been no'ted is the tendency 
of those in charge to institutionalize, their power and to resist changes 
which might alter their position.' Goldstein,/ et al (1977) describe how 
groups of terr desire evaluation research to satisfy external demands to 

something," but simultaneously are looking for the results to justify 
established policies and procedures. Decision makers, 1t seems, are more 
receptive to research conclusions that fit nicely into established policies. 

; ' ' •' . . ; 13 . . • '. ' 
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This tendency -and ability- of powerful units to maintain their position 
acts to slow down the process of change and may easily contribute to both 
a desire and an ability on the part of powerful subunlts to attend to se- 
lectively (or Ignore) utilization information, depending on their Interests 

These findings suggest several propositions with respect to the 
influence of evaluation units and the utilization of evaluation research: 

'Proposition l: Results of evaluation research is more likely to 
be used by decision makers when: 

a. the evaluation unit 1s powerful compared to the unit being 
evaluated. 

» • ■ ■ . 

b. the evaluation Information. 1s not available from other sources f 
i.e., the control of uncertainty 1s not subs titrable. 

c. the evaluation study, was on a program requiring a substantial 
proportion of the agency's resources; I.e., the project eval- 
uated. Is central to the agency's functioning, 

d. the criteria used for evaluation are criteria which the unit 
being- evaluated has accepted, I.e., there 1s agreement between 

' the evaluating unit and the one being evaluated on what 1s to 
be evaluated. . 

Organizational 'Goals, Incentives and Evaluation ' Q 

If, as suggested 1n Figure 1, the use of Information may be a function 
of the outcomes preferred by a decision maker, 1t 1s Important to consider 
» * how these preferences might develop, 'in an unconstrained setting, predic- 
ting the, preferences or tastes of decision makers is problematic. With the 
exception of some global notions of hedonic preferences or the idea that, 
1n general, people will choose to be "better off" than "less well off," . 

» 

it fs virtually Impossible to predict a priori what set of outcomes a 
given decision maker is likely to value. In organizations, however,. - 
preferences are far more, predictable. Decision makers exist 1n settings 

' * * • 
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which act to constrain both the range of outcomes which might be preferred 
in situations where there is a choice and the preferences for particular 
outcomes within this reduced set. These contextual constraints c\re ty'pi- 
cally imposed through systems of goals, control systems and incentives. 
Decision makers are usually constrained first by the nature of the job; 
that is, organizational or subunit goals are imposed along with the respon- 
sibility to work toward attaining these goals. To ensure that this is ac- 
complished, some, form of control system is typically used which allows 
superiors in the hierarchy to monitor the achievement of subordinate de- 
cision makers. Upon discovery of a variande, sanctions can be applied. 
Thus, the organizational context acts to constrain decision makers to pur- 
sue a limited number of goals and to reward or punish them for success or 
failure. ~ . ' "* 

Without digressing into a lengthy discussion of goals and goal formu- 
lation processes (cf . , Mohrn, 1971; Perrow, 1970), it should be noted that 
this process involves the translation of often vague, non-operational, long- 
term goals Into more specific, short-term outcomes. This activity may, as 
has been empirically demonstrated (e.g., Pfeffer & Salancik, 1§78)," be 
accomplished through a political process of negotiation, bargaining, co- 
opting, coalition formation, Aand the garnering and application of power. 
The process may also result in the adoption of symbolic and non-operative 
goals whose purpose is often.to satisfy external audiences and to provide' . 
for minimum constraints on organizational actors (e.g., Perrow, 1970; 
Simon, 1964). ■ Meyer and Rowan's (1977) Interpretation of the symbolic nature 
of the operation of schools is an illustration of these goals. The concern 



in many public organizations with accountability is another goal, per- 
haps more relevant as a symbolic effort to reassure taxpayers, than an 
operative goal relevant to effective and efficient organizational • function 
1ng. 

With Respect to evaluation, Orlans (1973) has noted that the wording 
in enabling legislation is often written 1h vague terms in order to gain 
sufficient backing ^rom legislators to Insure passage. However, since pro 
gram goals are not well-defined, 1t 1s difficult to develop mefsurable'" 
criteria, despite the fact that the legislation often mandated an eval- 
uation component. The lack of wel 1 -sped fled goals may encourage the 
political bargaining over criteria described by Pfeffer and Sal and k 
(1977). It may also result in- evaluations which opponents claim are not 
appropriate. Program Head Start legislation, for example, offered no de- 
lineation of goal priority or concrete objectives. The result was that 
results- of evaluation studies which showed no Impact of the program were 

often Ignored by decision makers as Inappropriate or assessing only a part 

. «•• •■• . 

of the program's intent (Gordon ~& Morse, 1975). 

The result of the translation of organizational goals Into short- 
term operative subunlt goals, while perhaps an Imprecise and ambiguous 
process, has several Important ramifications for'an understanding of the 
use of Information by organizational decision makers. First, even when 
the goals are somewhat ambiguous, objectives act to focus the attention " 
and efforts of. those responsible. Second, although the goal may be Im- 
precisely defined, control, systems are almost always developed which In- 
clude measures of objective outcomes. These measures of observable 
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outcome act as highly salient foci against which performance may then be 
assessed. A large and persuasive body of research has documented the 
. motivating effect of setting simple goals (e.g., Latham &Yukl, 1975). 
For example, "studies of budgeting (e.g., Wildavsky, 1974), public em- 
ployees (Blau, 1964), students (Baum '& Youngblood, 1975), homeowners 
(Erez, 1977)» have all demonstrated that the mere setting of goals can 
act to direct behavior. Other research has also demonstrated that once 
an Individual becomes committed to an outcome through, public actions, 
there is a tendency to continue to pursue the same goal, even 1f pre- 
ceding efforts met with failure (e.g., Salandk, 1977; Staw & Ross, 
1978). These findings suggest that 1n an organizational context, de- 
cision makers will probably develop rather explicit preferences for out- 
comes.. . * " 

Finally, and most Important, these preferences are solidified 
through the operation of- organizational Incentive systems: decision 
makers are rewarded for pursuing certain ends and punished for others. 
The pervasive Impact of Incentive systems should not be underestimated. 
Steve Kerr (1975) provides "a number of Instances 1n which employees be- 
haved 1n seemingly contradictory ways, often appearing to dS the opposite 
of espoused goals. For example, directors of orphanages were found to 
establish policies that worked against the placement of children 1n foster 
homes, and acted to keep them In the brphanage; universities routinely 
establish Incentive 'systems that militate against encouraging high quality 
teaching; sports teams often reward individual performance when a team- 
oriented effort 1s required. Upon analysis, the results were seen not 
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to be contradictory but to be entirely consistent with the incentive 
systems; that is, people in organizations typically do those things for 
.which they are rewarded. In Kerr's (1975) example, for instance, orphan- 
age directors who succeeded in placing children in foster homes would 
be "rewarded" by cuts in staff, resources, and prestige among peers. In 
a direct test of the hypothesis -that decision makers would respond to the 
operative control system, Harrell (1977), demonstrated that subordinate 
decision makers would follow their superior's lead and make decisions 
using similar criteria. This effect persisted even' when superiors began 
making decisions which were contrary to the official policy. 

Obviously, the goals of survival and obtaining resources are probably 
more important to program 'administrators and staff than to independent 
evaluators. Information which suggests that the program is not effective 
or efficient is likely to result in negative sanctions against those run- 
ning the program. . Clearly, there may be conflicts between the program 
adminstra tor's goal of survival and the evaluator's' jjrial of an accurate 
assessment. The incentives for those with vested interests in the program 
may be to devise strategies to minimize the collection, dissemination, 
, and use of such unfavorable information, "even if such actions go against 
some* "objective" assessment of overall worth. 

These findings are applicable in understanding how. decision makers 
develop preferences for outcomes: subunit goals are made salient, mea- 
surable and rewarded, control systems provide feedback and sanctions which 
act to focus attention on achieving certain ends, failure to attain the 
desired outcomes may result in the loss of important organizational rewards 
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•such as promotion, pay, and status, as well as actual punishments such 
as demotion or termination. As individuals in organizations become more 
committed and less able to leave the organization, the importance of the 
outcomes 1s increased. Since there is seldom complete consensus among 
individuals and subunits on the goals or outcomes to be sought, the possi- 
bility of conflict and competition arises. Under these circumstances, 
it is possible that one person's gain is another's loss; hence, it is un- 
likely that decision makers will easily give up preferences for certain 
outcomes if it means personal loss. 

Some indirect evidence of these effects may be seen" in a review of 
93 evaluation studies by Gordon & Morse (1975). These authors found that 
evaluation researchers with some affiliation with the project being eval- 
uated were far more likely to report the project as successful than were 
non-affiliated evaluators (52 percent to 14 percent). Similarly, there 
was a greater tendency for noh-afflliated researchers to report program 
failure than were affiliated evaluators (32 percent to 14 percent). Wh.1le 
not suggesting directly that affiliated evaluators deliberately biased 
their studies, thi results are consistent with the hypothesis* that Incen- 
tives may lead to conscious or unconscious biasing of information. 

The set of acceptable or preferred outcomes on which a decision maker 
1s likely to focus 1s, 1ri part, made salient through the specification of 

a control system which assesses attainment of the assigned goals, incentive 

• ' . / ■ 

systems which sanction certain actions, and the committing effect. of pre*- 

vious behaviors. These factors, suggest the following proposition: 



Proposition 2: Results of evaluation research are more likely to 
be used by decision makers 0 when: 

a. the evaluation Information can be used to assess achievement 
of quantifiable goals. 

b. the evaluation Information 1s fed Into a well-articulated and 
operating control system which Includes an effective set of 
Incentives; I.e. , measured performance can be sanctioned. 

c. the evaluation Information does not recommend actions which 
are Incompatible with the existing control system's ability 
to monitor and sanction. 

l 

> d. the evaluation information results from studies performed by 
those who have a vested interest in the continuation of the 
project. 



INFORMATIONAL IMPACTS ON DECISION MAKING 



As suggested by Figure 1, Information plays a central roJe 1n the 
decision making process. Before information can have an Impact, however, 
it must reach, and be processed by, the relevant decision makers. In an 
evaluation context this means that the 'evaluation report must not only be 
available to users but must also be read, understood, believed, and 
acted upon. Failure by decision-makers either to obtain the Information 
or to process it, can result 1n non-ut1Hzat1on, This suggests the need 
to examine two separate steps 1,n the dissemination process: (1) how de- 
cision makers acquire information, and (2) now material ^processed cog- 
nltlvely, A substantial body of research 1s available which suggests the 
limitations on and biases of organizational decision makers with respect 
to information acquisition and use. 



Information Acquisition 

Accessibility of Information. A large number of laboratory studies 



20 

C5 



of information and decision making have documented the intuitively reason- 
able conclusion that better quality information is generally associated 
with improved decision making performance (e.g., Porat & Haas, 1969; 
Streufert, 1973). Unfortunately, several authors have also noted that • 
the majority of these studies may be overly . structured when compared to 
the real-world situations they are meant to model (e.g.-, Connolly, 4977; 
Winkler & Murphy, 1973). Thus, while we have support for the importance 
of information in decision making, "we need to examine the process by which 
decision makers acquire information before concluding that the mere avail- 
ability of better quality information insures improved performance. " 

When we examine this evidence, doubts about the information-decision 
making link emergl". Decis< makers of many types appear to be noticeably 
biased 1n their procurement information. For example, 1n a direct test 
of the Impact of accessibility and quality of Information on Information 
source use, O'Reilly (1979) found that although decision makers recognized, 
Information sources of high quality, they used sources which provided lower 
quality Information but were more accessible. O'Reilly (1979) explains 
these results 1n terms of the costs Involved in obtaining Information' from 
less accessible sources. Given that the decision makers were under-time 
constraints and subject to numerous interruptions, it may have been that 
they were simply unable to seek out higher quality . information when 1t 
came from less accessible sources. Similar findings have been reported 
about the Information seeking behavior of physicians (Menzel & Katz, 1955), 
scientists (Gerstberger & AM en, 4969), policymakers (Clausen,' 1973), and 
managers (Mlntzberg, 1973). 



This bias towards accessible Information is also reflected in managers 
strong preferences for oral as opposed to written information (e.g., Dew- 
hirst, 1971) and for information from trustworthy or credible sources 
(e.g.,' Beach, Mitchell, Deaton & Prdthero, 1978; Giffin, 1967 Research 
in these areas has shown that managers typically preferj)Shorter, oral re- 
ports to longer written ones. Interestingly, there is also evidence^ that 
when obtaining information in this manner, managers may judge the validity 
of the information, not on the facts of the matter, but on the credibility 
of the source. This may lead to the acceptance of a piece of information 
as "true" or ""false" depending on how much the recipient trusts the sender. 
Clausen (1973), for instance, noted that Congressmen frequently cast votes, 
on legislation based not on an understanding of the deeper issues but on 
the advice of a trusted colleague. This behavior is not necessarily bad, 
but simply reflects the inability of a Congressman to be fully informed 
on all issues. * . ** 

The research on source credibility also suggests that it may be that 

it is the "saVeness 11 or trustworthiness of the source, more than expertise, 

which determines whether informatipn is believed (O'Reilly & Roberts, 1976) 

The ramifications of this finding for understanding the utilization of 

evaluation research may be important. Evaluation units, by their nature, 

. * • ^ x \ 

'may be perceived of as not sharing the same* objectives as other 

i.e., not being trustworthy in terms of source credibility. The 

' 1 

may be seen as objective rather than sympathetic; that lis, expert but not 
trustworthy/ Thus, there is some possibility that the information produced' 
may be used less by decision makers th^n information from "safer 41 sources. 
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David (1978), for example, found that the feedback of evaluation data 

which Included personal explanations by evaluation staff personnel was 

much more likely to be made use of. than a report that was merely delivered. 

Together, these. biases may .lead decision makers toward preferences 

for information from particular channels which they may characterize as 

accessible, trustworthy, and which provide condensed treatments of complex 

issues. Information from these sources, as will be suggested later, may 

be concrete and easily assimilated, but not necessarily detailed or of 

the highest quality. With respect to the utilization of evaluation research. 

this suggests the following proposition: 

Proposition 3: Results of evaluation research are more likely to 
. be used by decision makers 1f they are: 

a. readily accessible to final decision makers 

b. summarized 

c. presented. orally ' 

d. from a'source deemed as credible; I.e., "trustworthy. 
Information sign. An additional bias noted 1n studies of information 

.acquisition by decision makers 1s a tendency to avoid information which 
may suggest undesirable consequences. Janls and Mann (1977) offer several 
examples of this behavior describing how politicians and. policy makers, when 
faced with unpleasant alternatives, will avoid exploring ominous 1mpl1ca- 
tlons of desired courses of action-. - Several examples of this sort are 
available 1n studies showing that program administrators have exhibited 
tendencies to dismiss negative findings (e.g.» Borgatta, 1966; garter, 
1971). Qoldstein et al (1978, p. 33), report similar instances 1n which 
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evaluation results were selectively interpreted as offering support for 

a politically favored position. This willingness to avoid acquiring pe- 

• , • 

jorative information and to seek out or selectively process favorable 

i .. . . - 

items has direct ramifications for evaluation research. As suggested 

earlier, decision makers often have vested interests in certain outcomes. 
This may predispose them to seek out information which supports their 
position. David (1978) quotes- a decision maker as saying,. "I want in- 
formation to justify the expansion of the program" (p. 17). Bear and 
Hodun (1975),. for example, fotfnd that subjects were likely to recall 
items of. information that were confirmatory to their position rather than 
thos^e that were contradictory. Interestingly, this bias extended to re- 
calling some contradictory items as supportive and even to recalling 
mis Jng data as confirmatory for a preferred position. In an interesting 
laboratory study, Morlock (1967) demonstrated that it required •signifi- 
cantly leas information for subjects to arrive at a decision favorable 
to their position than to arrive at a decision considered to be* against 
their interests. In an organizational setting, groups of like-minded 
decision makers may exaggerate these biases toward selective perception 
and actually act collectively to censor or derogate information in oppo- 
sition to their desired ends. Janis (1972), labelling this process as 
"groupthink," provides a number of retrospective accounts in which groups 
acted to bolster desired opinions and exclude contrary ones. For Instance, 
decision making by President Kennedy's advisory committee during the Bay 
of Pigs. Incident was characterized afterwards by the suppression" of doubts, 
creation of feelings of unanimity and invulnerability, and an unwillingness 
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to risk conflict within the group. Johnson (1974), in a study of group 
decision making, provides a nice example of this tendency to seek unanimity 
and avoid conflict. She hypothesized that executives, would make a less 
desirable, but acceptable, short-run decision to avoid generating conflict 
with others in the group. Using 49 businessmen across ten situations she 
discovered that although subjects could identify the ideal , decision for each 
situation, their overwhelming tendency was to make a sub-optimal decision 
in order to avoid conflict. David (1978) again quotes a user of evaluation 
data as saying, "I look at test scores mainly to confirm my own impression.- 
If ttiey differ, my impression counts 11 (p. 16). 

These biases, that is, tendencies' to avoid information which suggests 
undesirable consequences, to seek out supportive information, to require 
less supportive information to arrive at a favored decision, and a desire 
to avoid making decisions which will generate conflict, suggest the fol- 
lowing proposition. 

Proposition 4: Results pf evaluation research are more likely to 
be used by decision makers if the information: 

a. is supportive of the outcomes favored by the decision makers. 
• b. does not lead to conflict among the set of relevant actors. 

Proposition 4 appears most appropriate when, as Jam's and Marin (1977) 

suggest, the -group is cohesive and members value their associations. Under 

these circumstances, it may be that group members will be more apt to sup- 

press conflict in the name of group unaniminjty. * When evaluation informa- * 

tion is available to competing factions or decision makers belonging to 

differing groups, the following proposition is suggested: 

Proposition 5: Results of evaluation research are more likely to 
be sought out and used by decision makers under competing or con- 
flicting conditions when: 1 



a. Information supportive of an undeslred position is available 
to actors seeking different outcomes, 

b. The decjsion making process is drawn out over a long period 
of time~ which allows alternative positions to be presented. 

Cormtunioation of Information. A final set of potential- biases on 
information acquisition by decision makers in organizational settings is 
related to the communication of information in organizations. Several 
authors have noted that communications in organizations may be withheld 
or distorted (e.g. , O'Reilly & Pondy, 1979; Wilensky, 1967). Subordinates 
have been found to be biased toward passing superiors information which re- 
fleets favorably on the subordinate and suppressing unfavorable information. 
O'Reilly (1978) discovered that subordinates who did not trust their supe- 
rior were willing to suppress unfavorable information even-if they knew 
that such information was useful to their boss. Other investigations 
have shown similar results. For instance, Petti grew (1973) documented 
how a single individual, acting as a gatekeeper for information flowing 
to a policymaking group, was abl$ to determine the outcome of a purchas- 
ing decision by carefully allowing only certain types of information 
through to the decision makers. Plott and Levi ne (1978) demonstrated how, 
through the arrangement of the agenda of a meeting, outcomes could be deter- 
mined in advance. Lowe and Shaw (1968) provided evidence that departments 
systematically inflate.and bias budget requests to support claims for in- 
creased resources. In a slightly different vein, Kaufman (1973) showed 
how subordinates learned not to pass certain items of information upward 
in the hierarchy because superiors, upon learning of tt\ese, would be re- 
quired to act in ways contrary to their self-interest. A classic example 
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of this system- Induced distortion 1s provided by, McCleary (1977) 1n. a 
study of how parole officers report clients' violations, observing that 
parole officers under- reported deviant behavior to their supervisors. 
As noted by Kaufman (1973), subordinates are often punished for accurate 
reports. In McCreary's s^udy, this resulted 1n Incidents ; be1ng reported 
only when the Information sent upward would resul't 1ri enhancing the sub- 
qrdlnate's career. 

Numerous other examples are available which attest to the fact that 
information Is often selectively filtered and distorted as 1t 1s communi- 
cated 1n organizations, (e.g., Allison, 1971; Janowltz & DeTany, 1957, etc.). 
Such filtering and distortion appe'ars to result, most often, from individ- 
uals or groups attempting either to gain desired outcomes (such as Increased 
resources or pjpwer) or to avoid losses of these. When considered in con- 
junction with the biases toward reliance by decision makers on short, oral 
rjeports from -trusted sources," the Impact of distorted Information 1s , 
heightened. 

■■■■ ' ■ * ' ■ 

Clearly i these biases have Important consequences for the transmission 

and use of evaluation research since Interpretation of the results of eval- 

uation studies may be biased either to support or refute a particular 

position. If, as March and Simon (1978, p. 165) note, "inferences are . ' 

drawn from a body of evidence and the Inferences, Instead 'of the evidence 

Itself, are then communicated," the opportunity for subtle distortion 1s 

magnified. Findings which have undesirable consequences may be withheld 

by superior gatekeepers also. Direct evidence of this effect is recounted 

by Coleman (1972) who describes how HEW attempted to minimize the Impact 



of an EEQ study because Its findings were Inimical to the Interests of 
some other HEW agencies. Here we find decisions being made to suppress 
or alter Information both to" minimize conflict and to avoid undesirable 
outcomes. 

It should be noted that not .all distortion in organizational communi- 
cation- 1s, necessarily Intentional. With any transmission from one 1nd1v1d 
ual to another there 1s almost always some degradation of ' the message (e.g 
see Campbell, 1958). Receivers of Information recall certain parts "of 
messages and forget or minimize others. This unintentional distortion, 
due to differences 1n cognitive tuning, may be Increased when the communi- 
cation occurs between groups who use different vocabularies, are sensitive 
to different goals and constituencies, or are-us1ng different criteria 
for determining what 1s Important. Janls (1972), for. example, describes 
how Information during Pearl Harbor was not transmitted because- senders, 
unaware of the broader picture," did not perceive certain information as 
Important. Evaluation researchers, whose concerns are often somewhat dif- 
ferent than administrators, may not be fully aporedatlve of the concerns 
of users of their studies. This last,_the use of a different vocabulary 
and the requirements for communication through several hierarchical levels 
may 'lead to evaluation reports which are seen by admlnstrators as not 
timely, too technical, focused on issues which aren't central , and genera 
ally .unconvincing (e.g. \jM kin, 1975; Cox, 1977). Such opinions on the 
part of administrators coupled with biases mentioned previously, such as 
a reliance on accessible and credible information sources, make it clear 
that evaluation information may not have the impact expected by evaluation 
researchers. 
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Acknowledging that Information may be blocked or altered during 

communication within organizations suggests the following proposition: 

Proposition 6: Results of evaluation research are less likely 
to be useful when: 

a. transmitted through several Intermediate links 1n a communi- 
cation network rather than delivered directly to the relevant 
decision makers. •* 

b. transmitters; of the Information are likely \o suffer personal 
' or organizational losses from the message. 

c. senders and receivers do not trust each other. 

d. the. Information 1s transmitted to decision makers 1n another 
agency or organization. 

d. the Information 1s transmitted to decision makers 1n a differ- 
ent functional group within the same organization. 



Information Processliq 

Until this point, our discussion has focused on biases which may 
affect the acquisition of Information by decision makers in'an organi- 
zational context. Once the Information has been acquired, hiwever, it 
may still be that individuals will fall to process 1t accurately. A 
diverse and well-developed body of research 1s available which documents 
these limitations 1n cognitive Information processing (e.g., Slov1c n 
Flschhoff & L1chtenste1n, 1977). Two general themes from this research 
are relevant to an understanding of the use of evaluation research by • 
decision makers: (1) factors which limit one's ability to assimilate 1n- 
formation,' and (2) processing strategies which may result in Inaccurate 

or misleading Inferences. . 

*.'• ' "'• - • * 
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Cognitive Limita on Information Prooeaaing. Early studies of human 
Information processing demonstrated unambiguously that only limited 
amounts of Information could be used by decision makers. Miller (1956), 
1n a classic study, demonstrated experimentally that short term memory 
1s limited 1n most people to seven "chunks" of Information plus or 
minus two, (a "chunk" being the largest single Item of Information rec- 
ognized by the processor) and that an Individual's ability to "chunk" 
information acts as a direct constraint on the Input of data. What 1s 
Important to us 1s that this fundamental physiological limitation appears 
to act as a constraint on all decision makers. Numerous studies, have 
demonstrated that physicians, stockbrokers, meteorologists, policymakers, 
and a variety of other "experts" all appear to use only a very limited 
number of cues 1n making complex decisions (e.g., Dawes & Corrigan, 1974; 
Slovlc et al, 1977; Slovlc & L1chtenste1n, 1971). Other research 1n this 
area has also shown that variables such as personality, cognitive stcuc- 
ture, and demographics are related to the ability to process Information 
(e.g.', Nystedt, 1972, Schroder, Driver & Streufert, 1965; Taylor & Dunnette, 
1974). Further evidence of limitations on the ability to process Infor- 
mation 1s available In studies showing the biased nature of memory and 
perception. Buckhout (1974), for example, lists three common sources of 
unreliability 1ft recall; (1) Insignificance 1n the original situation, 
i.e, , cues which were later deemed Important were not closely attended 
to at the time, (2) the degradation of Information. 1n memory over time, 
and (3) pressures and distractions on the Information processor Which re-' 
duce the, amount and accuracy of information used. 
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Each of these limits Is potentially Important for an understanding 
of how decision makers may use evaluation Information. For instance, 
given that decision makers are able to use relatively little information, 
1t becomes problematic to know which items of information of the total 
quantity available a decision maker will focus on and use. Studies 'of 
experts are consistent in finding that experts on' the same subject typi- 
cally use different information in making expert judgments (e.g., Slovic 
et al, 1977). This suggests that decision makers, when presented with 
evaluation research results, may interpret and weight the information 
differentially. Hawkins et al (1978), for example, in a drug evaluation 
study, showed that various actors were weighting information differently, 
according to the evaluation criteria they were using. Other studies have 
also demonstrated variations in preferences for types of information across 
decision makers (e.g., Kilmann & Mitroff, 1976), as well as in how stress 
reduces one's ability to process information (e.g., Wright, 1974). Since, 
as Mintzberg (1973) has shown, managers' work is characteristically frag- 
merited and subject to distractions and time pressures, 1t 1s likely that 
users of evaluation research will be unable to assimilate fully all the 
information contained in a report. Instead, users are likely to form 
overall impressions, subject to the biases mentioned previously, and to 
weight the results" accordingly. This interpretation is consistent with 
studies which show that with the passage of time details are forgotten, 
that „the reconstructed meaning is often less ambiguous than originally 
portrayed, and is then interpreted as offering support for a favored 
position (e.g., Buckhout, 1974; Ross, 1977); that is, information 
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contained 1n the original signal which 1s unfavorable 1s likely to be either 

forgotten or reinterpreted so as to minimize Its negative consequences, 

These and other studies of Individual limits on cognitive Information 

processing corroborate a reasonable but often neglected fact: memory 1s 

a selective and often fallible source of Information. The perceptual pro- 

cess of detecting and attending to stimuli 1s affected by factors such as 

stress. With the passage of time there 1s also a tendency to reorganize 

our "memory" Into coherent recollections by stripping away contradictory 

evidence and filling 1n any gaps with "constructed" facts. These limitations 

suggest the following propositions: 

Proposition 7: Only Umlted amounts of Information from an eval- 
uation study are likely to be used for decision making purposes. 

Propoaition 8: "Given the same, evaluation Information, different 
experts will' use different parts of the report 1n different ways; 
that 1s, judges will weight differently the same evaluation Infor- 
mation. 

Propoaition 9: 0ver f t1me, decision makers will be more Tfkely to • 
Interpret favorable Information from an evaluation study as less 
c ambiguous than originally perceived. 

Propoaition 10: Over time, decision makers will be more likely to 
forget unfavorable Information from an evaluation study or reinter- 
pret such Information as either Irrelevant (e.g., does not address 
the "relevant" question) or favorable. 

Selective Prooeaaing. Astde from these limits on decision makers' 
ability to process Information, there also exist biases 1n the manner 1n 
which Information 1s c*ogn1t1vely processed by Individuals. . Three of these 
biases have direct ramifications for evaluation research utilization: 
(1) selective perception, (2) self-serving biases, and (3) a preference 
for vivid, concrete Information for use 1n decision making. The first 



relates to the tendency, described 1n the previous section, for users to 
"rejnterpret" Information which has been acquired to fit preconceptions 
or to allow the user to maintain a consistent set of attitudes and beliefs 
about a given topic. This bias toward consistency Injected by the re- 
construction of facts, 1s dramatically Increased through the three selec- 
tive processing mechanisms mentioned above. 

Janls and Mann (1977) offer a number of excellent Illustrations of 
Instances 1n which decision makers either defensively avoided acquiring 
or processing unfavorable Information or bolstered their position through 
the selective acquisition and Interpretation of favorable data. This 
tendency may be seen 1n decision makers' willingness to overweight nega- 
tive Information when they desire to make a negative decision (Kanouse 
and Hansen, 1972). In these circumstances, when a decision maker desires 
to reject an* opposing view, judges have been shown to use whatever negative 
Information 1s available to say no. Selection Interviewers when presented 
with a large number of positive cues and very few negative ones have been 
shown to attend to the negative Information systematically and to use 1t 
to reject applicants even though the positive Information 1s far more 
potent. Miller and Rowe (1967), for example, found that when subjects 
were required to make assessment decisions, there was a significant tend- 
ency to be Influenced by negative rather than positive adjectives used to 
describe a candidate. Other corroborative evidence 1s available from 
studies ol personal perception among those who make Investment decisions, 
among "gamblers, and others (Kanouse & Hansen, 1972). When decision makers 
favor a position, the bias has been shown s to operate towards the selection 
of favorable information as well (e.g. , Mori ock, 1967). 
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It should be noted that this bias does n6t necessarily suggest that 
decision makers truncate their search for Information having once obtained 
data which can be used to support a desired position or, oppose an un- 
deslred one. The apparent tendency 1s to selectively seek out Information 
which bolsters one's position and avoid unsupportlve Information 1n either 
acquisition or processing, but hot necessarily to avoid searching. In 
fact, a number of laboratory studies have demonstrated an interesting 
propensity among decision makers to desire more Information than can be 
effectively used (Chervany & Dickson, 1974). The paradox 1s that decision 
makers appear to seek more Information than required, even to the pol^t of 
Inducing overload. While the overload may actually Impair performance, 
the additional Information has been shown to Increase the decision makers 
confidence (Chervany & Dickson, 1974; OSkamp, 1965). The net results may 

be that decision makers arrive at poorer* decisions but are more confident 

■ 

1n their choices. 

Thus, 1t may be that decision makers will selectively seek out Infor- 
mation which supports or opposes a position, acquire as much of this In- 
formation as possible, and be Increasingly confident 1n their decision al- 

though such decisions may be substantially biased. Mel tsner (1976), 1n 

* ,. 

a, book on policy analysts 1n bureaucratic settings, makes a relevant dis- 
tlnctlon between two categories of Information sought by decision makers; 
that 1s, Information used to make decisions and Information Used to support 
decisions. The latter category 1s indicative of the type sought by de- 
cision makers to justify a position. Meltsner describes at some length 
how 1t 1s not uncommon for decision makers to hire outsia^consulting 
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groups to do evaluation studies, not to be used for decision making pur- 
poses but solely to provide credible Information which supports a decision 
which has already been made, This Idea 1s directly related to the earlier 
discussion of Information as a political resource^ and suggests the fol- 
lowing proposition; 

ProvoHHon ii: Results of evaluation research are more likely 
to be used If portions of the study can be selectively Interpreted 
as either supportive of a desired position or unsupportlve of op- 
posing positions. 

Self-Serving Biaaaa. Aside from propensities to perceive and process 
Information selectively, decision makers have also been shown to engage 
consistently 1n what has been labelled "self-serving" biases; that 1s, 
researchers have noted that Individuals often view themselves more favor- 
ably than seems objectively warranted (e.g., Miller & Ross, 1975). Thus, 
for example, people have been shown to overpredlct their gambling successes 
consistently (Blasovich, Glnsburg & Howe, 1975), production managers over- 
predict their performance (Kldd & Morgan, 1969), and corporate presidents • 
overpredlct their firm's success 1n meeting competition (Larwood & Whltaker, 
1977). Aside from this future-oriented optimism, Investigators have also 1 
shown that, 1n retrospect, members of successful groups see themselves 
as more responsible for their groups than do members of groups that have 
failed. Schlenker and Miller (1977), for example, found that members of 
groups tha>%ad failed assigned less responslbmtF'for the group's poor' 
performance to themselves than they typically assigned to any other member 
of the group. 

In organizations, such pervasive "self-serving" biases have Important 
ramifications for the acquisition and interpretation of information. 

« v . - 

ii * • 
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These biases become especially' Important when participants are linked to,, 
or are responsible for, previous decisions to allocate resources. Under 
these conditions, decision makers may be committed to a particular program. 
Staw and his colleagues (e.g., Staw, 1974; Staw & Fox, 1977) have demon- 
strated how commitment to a course* of action may result 1n escalating 
commitments of resources to falling projects. Staw and Ross (1978), for 
Instance, demonstrated that policymakers who had allocated resources to 
projects- which subsequently failed for reasons that they should have fore- 
seen, were more likely to devote even more resources to the project 1n 
successive time periods than were decision makers who had sponsored suc- 
cessful projects or whose projects had failed for exogenous reasons beyond 
the decision maker's control, Th1 i example 1s similar to foreign policy 
failures si/ch as' the United States' Involvement 1n Vietnam. Staw and Ross 
quote George Ball (1965), who 1n the early years of the Vietnam War stated 
that, "Once we suffer large casualties . . our Involvement will be so 
jreat that we cannot —without national humiliation."- stop, short of achieving 

* » ■ 

3ur complete objectives." These examples are disturbingly similar to cer- 
tain educational programs which persist 1n spite of evaluation research 
rtrtch documents their failure to achieve stated objectives. - % ' 

Two possible ^reasons for, such evehts can be offered. First, the stated. 
>bject1ves of the program which are evaluated and found wanting are not > 
representative of either the true objectives of the program (for example, 
*hen a program 1s established for political purposes and evaluated on edu- 
:at1onal attainments),' or the evaluation 1s done on only a subset of the 
total program goals. Or, second, evaluation Information attesting to the 



- • > 

program's 'Inadequacies' 1s Ignored. It 1s 1n this latter Instance that 
the effects of commitment described by Staw may be linked with self-serving 
biases. 

■ ,i* ■ • * 

We only speculate that the ability which Individuals exhibit to be 

over- optimistic about future events predisposes them to commit themselves 
to courses of action. Further, when cohesive groups are Involved, there 
may be an "even greater impetus to choose risky options. Once committed, 
the selective perception biases described earlier can act to provide in- 
formation supportive of the original decision. Hal bers tarn (1974) provides 
numerous Illustrations of ■ how Robert McNamara and others engaged in this 
act1v1t1y during the war 1n Vietnam. Gouran (1976) provides similar ex- 
amples showing that Nixon and his aides persistently discounted the 1mpor- 
tance of evidence during the Watergate cover-up. Biases 1n selective 
perception allow the parties Involved to choose Information, as suggested 
In Proposition 11, which either supports the alms of the programs or re- 
buts opposition claims. Self-serving biases may also act to allow for 
the development of a false consensus or the Illusion that' their behaviors 
and choices are common and appropriate while opposing responses are un- 
common and not widely supported. These self-serving biases also act to 
make the decision maker reluctant to abandon a chosen course of action. 
As long as information 1s available which can be Interpreted as support- 
ive of a gfven position, the bias on the part of the central actor will 
be to focus -on this corroborative Information. As Pfeffer and Salandk 
(1977) have shown, when ambiguity exists, particularistic criteria can 
be used by decision makers; that is, unless a widespread consensus exists, 
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1t 1s possible for opposing decision makers to argue for their positions 
and selectively to use Information to support thejr claims^ Given that 
evaluation 1s often prescribed p'recisely because the situation 1s am- 
biguous with respect to, a given program, it is obviously the case that 
the selective processing of Information will occur. Under these circum- 
stances, Individuals are likely to be Involved and committed to particular 
points of view and self-serving biases will be operating. This Impact 
may be heightened since the results of evaluation 0 research may also be 
ambiguous. I 

A final effect of selfr-servlng biases may be seen should a program 
be declared a failure, either for political or substantive reasons. 
Under these circumstances, self-serving biases may allow Involved partici- 
pants cognl t1 vely to reconstruct their involvement and devalue their re- 
sponsibility for the failure; selective perception may act to focus on 
exogenous events which explain lack of success in terms of others' actions 
and unforeseeable events. Thus, the operation of self-serving biases v 
suggests the following propositions relevant to an understanding of the 
use of evaluation Information. 

Proposition 12; Responsibility for a program results 1n Increased 
commitment on the part of decision makers and Increases the likeli- 
hood that the results of evaluation research will be* used when: 

- a. the evaluation Information can be interpreted as favorable - 
or, supportive of the program. 

b. the evaluatjon Information is ambiguous and can be argued as 
not reflective of the overall scope of the program. 

c. no strong consensus exists' as to the specific goals of the 
program such that some evaluation information may be Inter- 
preted as favorable to some parts of the program. 



Proposition 13. Results of evaluation which are unfavorable to 
a program are morejlkely to 'be used when: * 

a. therejexlst a set of actors with objectives contrary to 
thos^r associated with the evaluated program. ' 

b. resources are scarce, so that the competition for resources 
1s greater. 

Ab$traot Versus Concrete Information. Typically evaluations are con- 
ducted to assess whether a program meets Its goals, 1s successful compared 
to other projects, or 1n order to provide for the feedback of Information 
to Improve performance 1n succeeding time periods. Underlying these reasons 
1s the Idea that evaluation Information will allow decision makers to de- 

■i 

rive Inferences about causal relationships (for example, between educational 
Interventions and student achievement). An Important postulate of this pro- 
cess holds that causal explanations will be Influenced by consensus 1nfor- 
matlon, I.e., Information concerning base rates and how a given project 
fares compared to the base rate. Attribution theory researchers (e.g., 
Nlsbett, Borglda, Crandall & Reed, 1976) have drawn attention to the sub- 
stantia^ amount of evidence which has failed ta support the postulated 
effect of conserisus Information. For Instance, Tversjcy and Kahneman (1973) 
have demonstrated this point by asking subjects to judge 0 the probability 
that a target individual, described 1n a brief personality sketch, was an 
engineer, given: ^ 

(a) that he was drawn from a population of 70 engineers and 30 
lawyers, or 

(b) that he was drawn from a population of 70 lawyers and 30 
engineers. . " 

Knowledge of the population base rate for occupational categories had no * 

effect whatever on judgments of the probability that the target Individual 



was an engineer. Instead, subjects relied exclusively on the personality 
sketch and based the decision on the degree to which the description 
fitted the stereotypic engineer, or lawyer. Numerous other examples are 
available" which demonstrate that decisionmakers, and even scientists 
familiar with statistics, habitually Ignore information about the popu- 
lation and draw recklessly strong Inferences about the underlying popu- 
lation from knowledge of a very small sample. Kahneman and Tversky (1973) 
refer to this as the "law of sm^ll numbers." 

Observers offer some reasons why even expert decision makers ignore 
base rate or consensus information: Kahneman and Tversky, for example, 
speculate thit people may not know how to combine base rate information 
and, therefore, Ignore U, Nlsbett, et al, (1976) propose that base rate 
Information by Its very nature, 1s abstract and' pallid, and may simply 
lack the force to persuade subjects' to attend to h and to use 1t. Out-' 
siders, 1t seems, are unmoved by dry, statistical data dear to the hearts 
of scientists and evaluators. As Bertrand Russell observed, "Popular In- 
duction depends on the emotional interest of the instances, not upon their 
number'^ (1927, p. 269). Individuals respond to vivid, concrete information 
and ignore abstract datat. 

Nlsbett, et al. (1976);offer several examples of this tendency. Con- 
sumers have long ignored medical advice to quit smoking and safety advice 
to fasten seat belts. Such appeals typically report numbers such as the 
probability of being a victim and are largely ignored. Yet when a highly vis 
ible and concrete incident occurs which people can focus on, results are 
often dramatic. For example, medical exhortation on the value of early 
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detection of breast tumor long went virtually unheeded- Yet, the waiting lists 
at cancer detection clinics became months long after television reported 
the mastectomies of Mrs. Rockefeller and Mrs. Ford. In an Interesting 
experiment, Borgida and Nlsbett (1977) provided prospective students 
with course evaluations based on ratings of students who had previously 
taken the courses. This Information hacl little impaction course choices. 
In contrast, brief facerto-face comments about the courses had a sub-, 
stantial impact on course choices. Other studies have noted how vivid 
information which is non-diagnostic, that is unrelated to the decision 
to be made, may have an Impact on the choice (Troutman & Shanteau, 1977). 

Conslder^the ramifications of these biases for the use of evaluation 
information. The typical evaluation report is a document which relies 
heavily on the statistical analysis of data and variations from mean levels 
of performance. The essence of such a report 1s on base rates and varia- 
tions from 6 the mean. , Complaints by consumers of such reports that the In- 
formation Is 11 not helpful , M 1s " too dry," "relies too much on statistical 3 
analysis,' 1 or "doesn't (Jet at the real problem" may, in fact by symptomatic 
of decision makers 1 Inability to use abstract Information. Orr the other 
hand, users of evaluation Information often focus on a single, concrete, 
often dramatic, example even though the chosen example may not be repre- 
sentative of the larger picture. Patton (1978}, in his discussion of the 
-i "... / 

meanings of ^valuation data, calls for evaluation reports which represent 
something meaningful < to the identified information users, including efforts 
to reduce the mysticism of scientific jargon and to increase the face 
validity of measures. Clearly, concrete and vivid examples which are 

* 7V / " . 41 1 



accurate representations of the underlying results are likely to be accepted 
and remembered by decision makers more easily than compilations of statis- 
tics. Meltsner (1976, p. 234), reports the advice of a chief federal 
analyst that when writing two-page summaries of reports, it 1s Important 
to "sprinkle them with juicy punch lines that will catch the readers' In- 
terest." Such pragmatic advice reflects the fact that not only are de- 
cision makers busy enough, not to want to read reports, but they are also 
more likely to remember, and therefore more likely to use, vivid 1nfor- 
matlon. Another analyst reported that half his time was" spent as a "re- 
write man" trying to translate statistical material Into a form which 
would be meaningful to the President and White House staff and agency 
heads. These observations underscore the bias people have toward concrete 
Information and suggest the following proposition: , . . 

Proposition 14: Results of evaluation research are more likely to 
be used when vivid, concrete Illustrations of the conclusions are 
* - available. . " r 

SUMMARY AND CONCLUSIONS * 

. 0 

Our treatment of the utilization or non-ut1Hzat1on of evaluation 
Information by "organizational decision makers began with a simplified 
model of decision making whose purpose was to highlight the Importance 
and potential Impacts of Information 1n the decision process. We then 
argued that, 1n organizational settings, it may be that rather than the 
decision making process proceeding from a problem to alternatives to a 
choice which optimizes, organizational decision makers have strong prefer- 
ences for certain outcomes and act, 1n their information search and 
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processing, o 1n ways calculated to maximize the attainment of desired ends. 
This view recognizes that Information 1s only one commodity which may 
hejp or hinder goal attainment. This view also recognizes the fact' that 
some choices are unacceptable to certain decision makers, regardless of 
the net benefit to the larger collective. Knowledge does not necessarily 
equate with action. 

Given the political process through which goals and objectives are 
negotiated among groups of* organizational participants (e.g. , Pfeffer & 
Salandk, 1978), universal agreement on any allocation of resources 1s 
unlikely. This lack of consensus makes the process of organizational de- 
clslon making a political one, often characterized by conflict and dis- 
agreement. In order to achieve a semblance of rationality, 1f only to 
satisfy constituencies outside the. organization, we argued that procedures 
were established which give the appearance of comprehensive rationality, 
but, 1n fact, may be more symbolic than real. Within the bounds of this 
J, procedural. rationality" we argued that Individual decision makers were ' 
"contextually rational," that 1s, attempted to maximize goal attainment 

given a set of situational organizational and Individual constraints. / 

/ ■ ■ *. / 

Thus, 1t may be^that the requirement for evaluation is a manifestation 
of the need for procedural rationality 1n public management while the 
actual use of evaluation Information 1s subject to the contextual ration- 
ality of relevant dec1s1 6a makers. The fact that some evaluations are 
conducted/ as pro forma arrangements with funding agencies to Insure com- 
pliance with Regulations,, but not Intended for use 1n decision making, 
can be Interpreted as an example- of this procedural rationality. The . 



fact that other evaluations are conducted for support of previously made' 

decisions may be an, example of the contextual rationality of decision 

/ 

makers. • / 

•• ••■ 

Since the dedsl on-making process as Illustrated 1n Figure 1 1s. an 
Interactive one, and since the argument 1s that much of the Information 
manipulation stems fnom the preferred set of outcomes, it 1s Important 
to consider how these preferences are developed as .well as how Infor- 
mation 1s used. We proposed that two primary sets of constraints were 
relevant. First, in organizational settings, decisiqn makers are seldom 

Indifferent about outcomes. Rather, goals are assigned, for example, i 

... / 

profit or cost margins, and control systems established to monitor and 
sanction responsible Individuals. Power, or the ability to Induce other 
groups or Individuals to behave 1n prescribed ways, becomesan Important 
consideration for goal attainment. The effect of these variables on de- 
cislon, makers is to make both salient and desirable a limited set of out- 
comes. These constraints, when coupled with potential loss of personal 
rewards such as status,- promotion, social approval, and money fc act to 
commit decision makers to certain outcomes. 

Once committed, we argued that decision makers were then potentially 
subject to biases 1n both the acquisition and processing of information 
for use 1n decision making. Evidence was .cited which demblstrated that 
decision makers were biased 1n their search for Information, preferring 
accessible information which supported their preferences rather than con- 
trary Information, even if such Information was of higher quality. Fur- 
ther, evidence was also available which documented how commitment to 



certain desired outcomes was associated with the distortion of Informa- 
tlon 1n organizational communication. Hence, 1t may also be that the 
Information available to unbiased decision makers may, 1f 1t has been 
transmitted through an#organ1zat1onal hierarchy, already contain In- 
accuracies or distortions. •• 

The Information processing of decision makers was also considered 
as a potential source of non-ut1Hzat1on of evaluation Information. In 
pursuing desired outcomes, decision makers were often shown to perceive 

and Interpret Information selectively. It has also been shown that human 

- - . ' - • * 

Information processors do not deal well with dry, statistical data, but 

prefer more vivid, concrete examples, even though such Information may 

be Inaccurate eadlng. These biases may be Important since eval- 

uation information 1s typically quantitative and statistical. It was 

argued that the combination of decision makers' selective perceptions of 

supportive Information and general preference for vivid examples, biases 

them away from the use of evaluation Information unless such Information 

1s supportive. _ . 

§ 

The joint effects of the situational and Individual constraints on 
Information use by decision makers 1s outlined 1n Figure 2. Context 
variables such as Incentive systems, group. norms, and organizational 
structure may act- to affect the Information which Is available to a "de- 
cision maker. Context variables, as well as the manner 1n which 1nfor- 
matlon 1s processed cogn1t1vely, may also act to affect Individual pref- 
erences for certain types of Information. In turn, these variables may 
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determine how and what Information Is used by decision makers. 0f 

I" 

■ Insert Figure 2 Rt>r*> 

When evaluation research 1s considered from the perspective developed 
1n this paper, and subject to the* constraints presented 1n Figure 2, 
several observations about utilization of evaluation Information are note- 
worthy. First, evaluation research, regardless of the rigor with which 
the study was conducted, 1s not likely to be regarded by decision makers 
as objective, nonpartisan Information. Rather, such Information will 
likely be viewed as useful to some Interested parties, threatening to 
some, and Irrelevant to others. The utilization of such Information 1n 
any decision making will probably reflect not any objective measure of 
quality of the ?*^*ch, but a number of factors Independent of the eval- 
uation study such as the degree of consensus or conflict among those In- 
volved 1n the decision process, the relative power of the participants, 
pressures on the primary decision makers, availability of other infor- 
mation, etc. In some cases, Individual characteristics of the decision 
maker may also determine the ut1l1zation/non-ut1l1zat1on of evaluation 
results. When studies are^ undertaken, not to provide Information for de- 
dslon, but as a means of increasing confidence 1n a position or for sym- 0 
boUc reasons,' 1t is unlikely that any direct Impact of the research will 
be observable. "J\iis does not Imply that such Information 1s not useful, 
only that Its function 1s not directly related to decision making. 

Finally, the propositions developed 1n, this paper are clearly tenta- 
tive and somewhat simplistic and only suggest possible hypotheses which 



might be tested empirically. Obviously, there are a large number of In- 
fluences on the use of evaluation research which have been omitted here. 

o * 

Therefore, what 1s proposed 1n this paper 1s not a well-art1c\ilatffl' theory 
of utilization of evaluation 1n decision making, but some tentative prpp- 
posltions based on previous research on organizational decision making 
and Information use. 

Future Research 

Based on the empirical and theoretical evidence reviewed here, sev- 
eral tentative recommendations' appear warranted. First, as suggested 
by David (1978), efforts to improve the quality of dataware not likely 
to affect local use of evaluation. Instead, local evaluators might, as 
suggested 1n Figure 2, be attentive to understanding the goals and con- 
straints operating on local decision makers. Rather than presuming that 
evaluation data are neutral and decision makers are rational, attempts 
might be made to present data 1n ways 1n which decision makers are likely 

» 

to feel -least threatened. These might Include strategies to reduce com- 
mitments to competing goals, to increase trust by receivers, and the 
targettin$ of data to specific goals. 

Actions such as these may Improve the* utilization of evaluation In- 
formation by decision makers. One may question, however, whether this 
effort 1s desirable. 8 As David (1978) discovered, one of the primary ' 
functions of local evaluation units 1s to meet Federal reporlThg re- ^ 
qulrements, not to serve as" a guide for program Improvement. Other sources 
of Information may be more useful in improving local programs. Perhaps 



local evaluation units should be serving more as a data collection and 
evaluation center for various Interest groups as well as- being involved 
with assessing student achievements; i.e., more' responsive 1n terms of 
the provision of Information than with evaluation per se. Certainly the 
evidence reviewed here would suggest that there are a plethora of "good" 

a 

reasons why Interested decision makers might not Ibcept Information from 
a local evaluation agency. If these evaluation units are to have an 
effect on decision making and program rev1s1on, H more attention needs to 
be paid to the existing constraints and biases on Interested participants 
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FIGURE 1 

SIMPLIFIED MODEL OF THE DECISION MAKING PROCESS" 
(Campbell, 1971) 
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CONTEXTUAL AND INDIVIDUAL VARIABLES AFFECTING THE USE OF 
INFORMATION BY ORGANIZATIONAL DECISION MAKERS • 
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